Skip to content

Phase 7.2 (spec first): the JS lexer's Verus spec and contracts - #178

Open
atiernan wants to merge 7 commits into
claude/js-jit-compiler-next-21ddc8from
claude/js-lexer-spec
Open

atiernan wants to merge 7 commits into
claude/js-jit-compiler-next-21ddc8from
claude/js-lexer-spec

Conversation

@atiernan

Copy link
Copy Markdown
Member

This is the first half of Phase 7 item 7.2: the JS lexer's spec and contracts, with no implementation. Rule 3 wants the spec reviewed before code is written against it. clazel-js-lexer stays at tier 0.

Stacked on #176. Merge that PR first, or retarget this one to main afterwards.

What's here

  • src/spec.rs: ECMA-262 2026's lexical grammar (§11.1, §12.2–§12.9, Annex B.1.1), written as Verus spec fns, pinned at commit 0248456c.
    • The five goal symbols are a parameter, plus an html flag for Annex B.1.1 that is false for modules.
    • It covers white space and line terminators, all comment forms, the hashbang, identifiers with escapes and their §12.7.1.1 early errors, reserved words, and punctuators.
    • It covers the full numeric literal grammar (separators, BigInt, non-decimal, legacy octal) and strings with their SV as UTF-16 (lone surrogates allowed).
    • It covers regexp literals (lexical part only), templates with TV and TRV, the longest-element rule, and skipping trivia with the newline flag ASI needs.
    • The file's header lists every modelling choice.
  • src/lemmas.rs: proven lemmas for the longest-element search, the trivia bounds and one-step progress.
  • src/examples.rs: positive examples evaluated in Verus.
  • src/contract.rs and src/api.rs: requires/ensures for next_input_element and next_token, stating three properties. No implementation exists yet, so they sit on a body-less trait, JsLexer, with no escape hatch.
    • Every input gives a result.
    • The tokens and the trivia skipped between them cover the input exactly.
    • The result agrees with spec.rs.

For the spec reviewer, first

  1. The longest-match and error rule (longest_len, lex_error). A numeric literal followed by an IdentifierStart is an error, so 3in and 1.e are errors.
  2. Unicode identifiers: ID_Start and ID_Continue are left uninterpreted beyond ASCII. The implementation must tie a generated Unicode table to them, and nothing is claimed about them yet.
  3. Annex B's comments, and the rule that the hashbang and HTMLCloseComment alternatives match only at index 0.
  4. TRV: the template escape rules are folded into utf16_of.
  5. identifier_early_errors_ok checks every code point, not only the escaped ones.

Left out on purpose

  • The numeric MV, which is 7.11's.
  • ASI itself, strict-mode early errors, regexp body parsing.
  • null, true and false come back as IdentifierName.
  • Negative examples.
  • A lemma that no two element kinds match the same range.
  • lemma_template_characters_cons, which Verus would not prove. The definition is unchanged.

Verification

  • Verus, on a fresh target dir in WSL: verification results:: 38 verified, 0 errors.
  • cargo xtask check: STATIC CHECKS PASSED. cargo build -p clazel-js-lexer, clippy -D warnings and cargo fmt --check are clean.
  • cargo xtask verify --changed was not run locally. The new vstd dependency changes Cargo.lock, which would make it re-check every crate. CI's full verification is the check, and its proof jobs need approval.

🤖 Generated with Claude Code

atiernan and others added 3 commits November 16, 2026 14:55
…first)

ECMA-262 es2026 at 0248456c: section 11.1, section 12 (goal symbols,
white space, comments, tokens, names, punctuators, numeric, string,
regular expression and template literals with SV, TV and TRV) and
Annex B.1.1 HTML-like comments, with the longest-element rule and the
two lexical early errors. Facts proven about it (longest-element search,
trivia skip bounds) and evaluated examples. Records one erratum (the
missing MV of the escape 0 before 8 or 9). Tier stays 0: no implementation.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ken (7.2)

Contract predicates in terms of the spec (totality, span coverage,
agreement, the line-terminator flag) and a contract-only trait with the
requires and ensures; no implementation, no escape hatch. Tier stays 0.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The pinned spec does define the MV of LegacyOctalEscapeSequence :: 0
(0) in the MV clause of §12.9.3.2, so nothing was missing.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
CI's proof job requires a mutations.json for every crate Verus checks.
Four mutations of the spec, each killed by a proven example: dropping
<LS> from LineTerminator, adding NEL to it, the wrong MV for `b`, and
dropping the recursive LegacyOctalIntegerLiteral alternative. Adds the
two examples they contradict (line terminators, hex digit values).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@atiernan
atiernan temporarily deployed to full-verification November 17, 2026 21:00 — with GitHub Actions Inactive
atiernan and others added 2 commits November 20, 2026 19:40
…e them

CI's mutation step (first proof run of this crate) failed: the kill
certificates for hex-digit-value-lowercase-offset and
legacy-octal-two-digits-only "do not build", because Verus's compute_only
leaves char-to-integer casts such as `'a' as int` unevaluated
(`clip(u32, '0') <= 48` stays stuck). Every char-literal cast in spec.rs is
now the code point in hex (`'a' as int` -> `0x61`), as the specs on main
already write them (clazel-base64, clazel-cookie). The meaning is unchanged;
the mutation's find/replace text follows the new spelling.

Locally (Verus 0.2026.10.04): 40 verified, 0 errors; 4 mutations, 4 killed
by a theorem with certificates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@atiernan
atiernan temporarily deployed to full-verification November 20, 2026 19:53 — with GitHub Actions Inactive

This branch was previously deployed

1 inactive deployment
full-verification — d88158ac Deployed Nov 20, 2026 by atiernan via Approve the proofs #614
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant